Repository navigation
fix(storage): set GRPC_ALTS_MAX_CONCURRENT_HANDSHAKES=2 in benchmark runner - #18638
shradhakatyal wants to merge 1 commit into
Conversation
|
/gcbrun (512d4e7) |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request enhances the GCS DirectPath read benchmark by adding median and maximum throughput columns to the results display, and optimizing the benchmark execution script. Key script improvements include limiting concurrent ALTS handshakes, preferring modern Python versions (3.11/3.12), pre-compiling bytecode, and pre-warming gRPC channels. Feedback on these changes highlights alignment issues in the output table borders, potential type conversion errors when parsing throughput values, risks of bypassing dependency resolution with --no-deps during pip installation, and the suppression of stderr during the pre-warming step which hinders debugging.
|
/gcbrun (7d52fce) |
|
/gcbrun (d78d379) |
|
/gcbrun (2701e2e) |
2701e2e to
0137b2e
Compare
shradhakatyal
changed the title
|
/gcbrun (0137b2e) |
0137b2e to
5cf9e92
Compare
|
/gcbrun 5cf9e92 |
5cf9e92 to
00ff4d5
Compare
shradhakatyal
changed the title
|
/gcbrun (00ff4d5) |
00ff4d5 to
28bd981
Compare
|
/gcbrun (28bd981) |
1 similar comment
|
/gcbrun (28bd981) |
28bd981 to
7a165cb
Compare
shradhakatyal
changed the title
|
/gcbrun (7a165cb) |
1 similar comment
|
/gcbrun (7a165cb) |
This branch has not been deployed
1 participant
Summary
In
benchmarks-cloudbuild.yaml, the standing benchmark VM (gcs-benchmark-runner-us-west4-a) is cold-started immediately before executingrun_benchmark_tests.shwith 48 concurrent worker processes (PROCESSES=48).By default in gRPC,
GRPC_ALTS_MAX_CONCURRENT_HANDSHAKESis100per process (48 * 100 = 4,800potential concurrent streams). When all 48 processes simultaneously initiate DirectPath ALTS handshakes against the local GCE metadata server (169.254.169.254:8080, ~100 stream buffer limit), the metadata server drops streams withUNAVAILABLE: GOAWAY received; Error code: 11; Debug Text: Buffers full {http2_error:11}, stalling Round 1 from35.3s(5swarmup +30smeasurement) to41.5s–49.0sand dragging down the 3-round average throughput.Change
GRPC_ALTS_MAX_CONCURRENT_HANDSHAKES="${GRPC_ALTS_MAX_CONCURRENT_HANDSHAKES:-2}"inpackages/google-cloud-storage/cloudbuild/run_benchmark_tests.shso that 48 worker processes never exceed96concurrent ALTS handshakes against the local metadata server while still allowing each worker to handshake multiple DirectPath subchannels concurrently.